A ground-truth dataset and classification model for detecting bots in GitHub issue and PR comments

Bots are frequently used in Github repositories to automate repetitive activities that part of the distributed software development process. They communicate with human actors through comments. While detecting their presence is important for many reasons, no large and representative ground-truth dataset available, nor classification models detect validate bots on basis such a dataset. This paper proposes dataset, based manual analysis high interrater agreement, pull request issue comments 5,000 distinct accounts which 527 have been identified as bots. Using this we propose an automated model bots, taking main features number empty non-empty each account, comment patterns, inequality between within patterns. We obtained very weighted average precision, recall F1-score 0.98 test set containing 40% data. integrated into open source command-line tool allow practitioners given repository actually correspond